Generally, conventional test execution frameworks in a distributed environment include a central controller and repository that manages the tests, agents that can execute this test, and a live connection from the controller to each of the agents. Usually, the controller controls which agent or agents will run the tests, and then deploys the tests to the selected agent. As such, the agents normally require code from the controller to be able to execute tests and report the results back.
Several drawbacks have been noted in conventional arrangements. First, an agent generally needs to open up an appropriate port to permit controller access to the agent. In so doing, the agent opens an unsecured gateway to a test lab, inviting possible breaches by unauthorized parties. Further, a persistent connection is usually needed through such an open port, thus increasing the likelihood of an unauthorized breach as a function of the time involved.
Typically, an agent will also require logging libraries to be provided in order to implement a custom agent, whereupon mismatches (e.g., in the area of code or version) of libraries may well become an issue in implementing the custom agent.
Additionally, in terms of an especially significant drawback, since the controller controls which agent to use, it may at times be the case that a selected agent will be unavailable, thus causing a desired execution to terminate abnormally.
Conventional solutions have involved the use of a Rational® Test Manager® agent and an Eclipse® Edition TPTP (Test and Performance Tools Platform) agent. (® is available through International Business Machines Corporation of Armonk, N.Y. while Eclipse® Edition TPTP is provided by the Eclipse Foundation, an open source community [www.eclipse.org]) Custom programs may therefore be provided which assume the role of a server and prompt controllers to connect over a non-standard port. However, in such a setting, the medium which facilitates communication between the controller and the agents will typically use non-standard and/or technology-specific data dictionaries or schema. Additional problems can thus arise from this, as briefly outlined below.
Firstly, firewalls as normally employed in a restricted or secure test lab environment will place limits on the use of non-standard ports. Communication as just described may be entirely precluded. Secondly, to the extent that many test labs often sit inside a highly restricted (e.g., militarized) network zone, access to external networks or connections may well be highly limited as it is. Thus, it may be difficult if not impossible to gain access to common resources such as file servers, update servers and the like. Further, an agent writer will typically have to learn the specifics of both the execution tool and the controller, so that data can be effectively retrieved, and that execution and other tasks can be initiated from the tool and logged back to the controller. For its part, the controller typically accepts data only in its own specific, non-standardized format.